Based on Google's BERT pre-training model, RoBERTa includes additional pre-training improvements that achieve state-of-the-art results on several benchmarks, using only unlabeled text from the world-wide web, with minimal fine-tuning and no data augmentation. The Facebook team announced their work in a recent blog post as "part of Facebook's ongoing commitment to advancing the state-of-the-art in self-supervised systems that can be developed with less reliance on time- and resource-intensive data labeling." The team re-implemented Google's BERT neural-network architecture in PyTorch, made several changes to the model's hyperparameters, and trained the network with an order-of-magnitude more data and for more iterations. The model was evaluated on three common NLP benchmarks: General Language Understanding Evaluation (GLUE), Stanford Question Answering Dataset (SQuAD), and ReAding Comprehension from Examinations (RACE). RoBERTa outperformed BERT on these tests, and in some cases also outperformed the current leading model, XLNet.